Multispectral data acquisition system and method

ABSTRACT

A portable multispectral data acquisition system for use on a vehicle such as an aircraft comprises a plurality of gyroscope-stabilized remote sensing devices synchronized to simultaneously capture images of a common spatial area in both visible and invisible bands of the electromagnetic spectrum, a computer and digital recorder to record and correlate the captured images with temporospatial reference information, image processing software to stack or layer the images and to extract and compare the images in order to identify and filter hidden or subsurface anomalies that are invisible to the naked eye, and additional image processing software to orthorectify the images to a three-dimensional digital terrain map and to stitch adjacent images together into a mosaic. Methods for using the portable multispectral data acquisition system are also provided.

RELATED APPLICATIONS

This application claims the benefit of, and incorporates by reference,U.S. Provisional Application No. 60/488,826, filed Jul. 21, 2003, andentitled “Multispectral Data Acquisition System.”

FIELD OF THE INVENTION

This invention relates to remote sensing systems, and more particularlyto remote sensing systems that employ a plurality of sensing devices todetect both surface and subsurface features of a place, region orobject.

BACKGROUND

The most familiar system for detecting the surface features of aterrestrial surface or object is the human eye. But there is far moreelectromagnetic information about an object than meets the eye. It iswell known, for example, that all objects having a temperature aboveabsolute zero emit electromagnetic radiation. Hot objects predominantlyemit radiation at short wavelengths. Cool objects predominantly emitradiation at longer wavelengths. Objects also absorb and reflectincident electromagnetic radiation. The variation of an object'sreflectance with respect to the wavelength of the incident light helpsdetermine what an object looks like. Indeed, different materials andmolecular structures emit and reflect electromagnetic energy indifferent but characteristic ways depending on the temperature of thematerial or the wavelength of the incident radiation.

There are a variety of commercially available sensors and imagersoperable to detect various wavelengths and intensities ofelectromagnetic energy, both visible and invisible, and both reflectedand emitted, emanating from various objects and materials. Employingknowledge about known characteristic energy absorption and radiationpatterns for different surface and subsurface materials and molecularstructures, it is possible to use information collected from suchsensors and imagers to detect and identify surface and subsurfacefeatures of a place, region or object that are invisible to or hiddenfrom the naked eye. These hidden or invisible features are commonlyreferred to as “anomalies.”

Surface and subsurface imaging technology has a wide field ofapplications. Imaging technology can and has been used, for example, tosurvey fields to search for pipeline leaks. In U.S. Pat. No. 4,963,742,I describe an airborne multispectral survey system comprising up tothree infrared detectors and two video cameras. Using standard,commercially available video tape recorders, the system simultaneouslycaptured and recorded images in both the visible region and in selectedinfrared bands of the electromagnetic spectrum of the terrain below.After the recording, analysis could be performed by simultaneouslyplaying the infrared and video recordings to identify anomalies—areaswhere the infrared recordings detected temperatures outside of a giventhreshold and which could not readily be explained from the visiblefeatures of the terrain. That system, however, heavily on humanobservation and on-site inspection of anomalies. Moreover, given thelimitations of the system, it was difficult to separate anomalies ofinterest from “false” anomalies (anomalies not of interest). Therefore,there is a need for more sophisticated multispectral remote sensingsystems that facilitate improved anomaly analysis capabilities.

SUMMARY OF THE INVENTION

A data acquisition system is provided comprising a plurality of remotesensing devices. One embodiment of the data acquisition system comprisesa wide angle visible-spectrum camera, a narrow-field-of-viewvisible-spectrum camera, a short infrared wavelength imager, a longinfrared wavelength imager, a laser mapping or scanning device, anultraviolet imager, a broad-band linear sweep laser, a magnetometer, andmeans for simultaneously recording the information collected. Means areprovided for integrating captured frames into a composite image ormosaic; for overlaying and comparing images captured from one imagerwith corresponding (i.e., simultaneously captured) images from another;and for identifying and distinguishing surface from subsurfaceanomalies. In one embodiment, the image capture devices are mounted onan aircraft such as a helicopter, together with gyroscopes or othermeans for stabilizing the instruments during flight and means forsimultaneously recording referential data provided by a globalpositioning satellite system and an inertial reference system. In otherembodiments, the image capture devices are handheld or mounted onterrestrial, sea-borne, or amphibious vehicles.

The primary purpose of the data acquisition system is to acquireinformation in several different electro-magnetic bands or frequenciessimultaneously so that data analysis will allow direct interpretation ofeffect and reaction of varying radiation levels or norms that prove tobe valuable information, not available in any single wave length. Thedata acquisition system allows visual reference, as well as computergenerated interpretive uses, and both real time and recorded analysisand review.

The data acquisition system works by looking for a combination ofradiation levels. The resulting images allow a computer application ortrained analyst to see and identify varying geometric digital profiles,or signatures, that violate the logical geometric comparable normpatterns. The several different remote sensors of the data acquisitionsystem are trained on a common terrestrial target area, for viewingselected frequency bands of the electromagnetic field. Perspective dataacquired by the remote imaging devices are preferably compiled to createwhat would appear to be a multi-dimensional (or layered) logic array of“stacked” imaging data. Some of this data is reflective energy, whileother is radiated energy. Differences that appear between the variousimages that may be referred to as “anomalies.” These anomalies arecaused by variations in the reflective and radiated energies indifferent frequency bands. Often, the variations are foreign effectscaused by an influencing generator or absorbent of the naturalthermodynamic characteristics in the target area. Anomalies are alsocaused by changes in the density of target mass.

The data acquisition system is preferably operable to provideinformation including, but not limited to, changes in the absorptioncharacteristics of sub-surface mass, changes in geological massradiation caused by fluids or gasses below or above the surface targetarea, and changes in radiation influences on ground, water or othersurface materials. The data acquisition system also preferably provideshigh speed assessment of the effect and reactions of varying levels ofboth radiation and reflected energy of the “target” and the effectscaused by virtually anything within the radiation target area that wouldcause an abnormal radiation ambient or average deviation. Thus, asuitable embodiment of a data acquisition system includes the followingequipment: a visual spectrum camera of 0.2-0.90 microns with definedcolor separation; a 3-14 micron infrared imager with varying ranges andthe capacity of a variable or “floating raster” of defined incrementrange that can manually or automatically sweep the ambient range of thetarget area; an advanced cesium magnetometer array that provides earthmagnetic flux data; a specialized laser imaging system combined with aspectrometer that provides profile and other spectrographic information;a time code generator or a GPS signal to ensure that each frame orsector of data is clearly identified; data recorders; broadband colorCCD cameras (0.2-0.9 microns); lenses for the cameras; powerdistribution box and cables; and color monitors for real time dataprocessing and system status monitoring. Other technologically enhancedembodiments of the data acquisition system may also incorporate anultra-violet imaging system, a high-resolution variable spectrumsolenoid, and side scan pulse sonar.

The image analysis process is able to support capture and retention offiles that conventional CAD engineering programs can read, thus makingthe data more usable. The analyst is able to overlay image data ontoG.I.S. data. This enables optimal analysis, superior inventorying andprecise locating capabilities. Interpretive analysis may be carried outon numerous phenomena ranging from subsurface mass, ground wateractivity, leachate and pipeline leaks to the density changes invegetation where drought, disease or photosynthesis problems may exist.

I believe my invention to be useful for a wide range of applications,including detection of water and gas pipeline leaks, mechanical failuresin industrial machinery, and electrical insulation damage on utilitylines. Other applications include detection of crevasses, road surfacedelamination, dam and dike seepage points, gas leaks and gas seepage,and subsurface dump sites. The invention is also useful for land usestudies and regulatory-mandated environmental surveys. Environmentalstudy applications include vegetation discrimination, urban microclimatestudies, airflow effects on vegetation near water bodies, crop moisturestudies, animal behavior studies, glacial drift studies, estuarineecology studies, arctic geology studies, groundwater studies,identification of subterranean streams, hot spring detection, rivercurrent studies, thermal effluent discharge detection and mapping, stripmining studies, forest fire detection and mapping, geological mapping,and volcano studies.

These and other aspects, features, advantages, and applications ofvarious embodiments of the present invention will be readily apparent tothose skilled in the art from the following detailed description takenin conjunction with the annexed drawing, which illustrate the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of one embodiment of a multispectral dataacquisition system.

FIG. 2 illustrates an aircraft equipped with another embodiment of amultispectral data acquisition system.

FIG. 3 illustrates a top level view of an on-board computer andcontroller for a portable multispectral data acquisition system.

FIG. 4 is a view of one embodiment of an aircraft-mounted platform forcarrying two cameras.

FIG. 5 illustrates a small hand held controller being held by a personin an ultra-light aircraft equipped with a multispectral dataacquisition system.

FIG. 6 is a scanned image depicting a plurality of correspondingorthorectification points on a pre-orthorectified image and atopographic map to which the pre-orthorectified image is to be fitted.

FIG. 7 is a scanned image illustrating an infrared image that has beenorthorectified to fit over a visible image slice.

FIG. 8A is a scanned image of a series of logically stacked spatiallycalibrated image slices simultaneously captured from different remotesensing devices.

FIG. 8B is a line drawing illustrating a series of logically stackedspatially calibrated image slices simultaneously captured from differentremote sensing devices.

FIG. 9 is a functional block diagram of one embodiment of a method ofoperating a multispectral data acquisition system.

FIG. 10 is a functional block diagram of one embodiment of a method oforthorectifying and layering images captured with a multispectral dataacquisition system.

FIG. 11 is a functional block diagram of one embodiment of an automatedmethod of processing a combination of visible, infrared, andmagnetometer data to identify anomalies.

DETAILED DESCRIPTION

Turning to FIG. 1, an embodiment of a multispectral data acquisitionsystem 100 is provided comprising a plurality of remote sensing devices110, one or more controllers 185 operable to control the focus,direction, and other operating characteristics of the remote sensingdevices 110, an image and data processor 190, and one or more digitalrecorders 180. The digital recorders 180 comprise any suitable media fordigital recording, including but not limited to digital 8 format VCRs,flash memory, optical drives, and standard computer hard drives. Theimage and data processor 190 comprises one or more computer processorsand random access memory for executing software applications thatcorrelate, filter, and process the data recorded on the digitalrecorders 180. Although the invention is not limited to any onecombination of remote sensing devices 110, it preferably includes aplurality of passive and active remote sensing devices, including atleast one remote subsurface imaging device.

As shown and depicted in the drawing, the data acquisition system 100comprises a wide angle visual camera 115, a narrow field-of-visioncamera 120, a 3-6 μm mid-range infrared imager 130, an 8-14 μmlong-wavelength infrared imager 135, a three-dimensional narrow-bandlaser range finder or laser mapping device 140, commonly known by theacronym LIDAR (light detection and ranging), an ultraviolet imager 145,magnetometer system 150, and a 3-axis variometer 155. The dataacquisition system 100 as shown and depicted in the drawing alsocomprises a broadband laser emitter 125.

In operation, the remote sensing devices 110 are mounted on an aircraftor other vehicle. The digital recorders 180 and controllers 185 are alsopreferably carried by the same vehicle carrying the remote sensingdevices 110, but with the help of high-bandwidth wireless communicationlink, could be housed or mounted elsewhere. The image and data processor190 or some portion of it may also carried by the same vehicle carryingthe sensing devices 110 and digital recorders 180 and in real-timecommunication with the same. Alternatively, the image and data processor190 or some portion of it is housed or mounted elsewhere, such as aground-based data processing facility.

FIG. 2 illustrates an aircraft 210 equipped with an embodiment of amultispectral data acquisition system 205. GPS antenna 220 receives GPSpositioning data from GPS satellites 222-225 and transfers theinformation to an onboard computer 208. An inertial reference system 230is communicatively coupled to the computer 208 so that it can record theroll, pitch, and heading of the aircraft and determine the orientationof the aircraft 210. A gyroscope-stabilized platform 211 for carryingremote sensors is mounted to the aircraft 210. The platform 211 carriesa first visible-spectrum camera 212 having a wide angle field of view213 and a second infrared-spectrum camera 214 having a narrow field ofview 215. The objective focal axis (not shown in FIG. 2 butcorresponding to the direction to which the sensor is directed) of thefirst camera 212 is substantially parallel to the second camera 214, sothat the two cameras are trained on the same target.

The platform 211 (or another like it) also carries a laser emitter 216and a laser receiver/detector to 218 scan the surface of the terrainbelow to record elevation data. This laser emitter and detector pair maycomprise a standard range-finding LIDAR, a differential absorption LIDAR(DIAL) (which compares absorbed wavelengths with reflected wavelengthsto identify molecules with particular spectrographic signatures), aDoppler LIDAR (to measure the velocity of a target), or a device thatcombines the capabilities of all three. In pipeline applications, thelaser emitter and detector pair is preferably tuned to an infrared, mostpreferably the near infrared, frequency or range of frequencies tofacilitate detection of pipeline leaks and the size of the flume.

FIG. 3 illustrates a top level view of an on-board computer andcontroller 300 for a portable multispectral data acquisition system. Thecontroller 300 has a digital data recorder 310 to record images receivedfrom its imagers and detectors together with temporospatial informationreceived from the GPS satellites 222-225 and inertial reference system230. The digital data recorder 310 can also record verbal informationspoken by the person operating the system (e.g., the pilot or anon-board technician) while the images are being recorded. A video screen312 on the controller enables an operator of the system to view imagesreceived from one or more of the sensing devices in real time.

The controller also has a motion control interface 320 with switches,knobs, and a joystick to control the sensitivity of the gyros and thespeed of the servos and to actuate the servos that control theorientation of the sensing devices carried by the platform 211. Theonboard controller 300 also includes a camera setting control interface330 to control various settings of the remote sensing devices carried onthe platform 211. Such settings include the digital and optical zoom (orfield of view) settings, the camera focus, and, for any IR cameras, theemissivity setting. Other optional adjustable settings (not shown inFIG. 3) include the ranging and power levels for the LIDAR system andany magnetometers and the frequency levels of any scanners andoscillators.

FIG. 4 is a view of one embodiment of the platform 211 for carrying twocameras 410 and 420, which for purposes of FIG. 4, is referred to as acamera pack 400. Camera pack 400 comprises a gyroscope stabilizedaircraft mount 440 and cameras 410 and 420. The camera pack 400 includesa variety of servos that are used to both adjust the position of thecameras 410 and 420 and to adjust various settings of those cameras. Forexample, one servo controls a pulley 424 and adjusts the focus of thelens 422. Another servo turns a pulley 434 to adjust the camera angle(i.e., the angle of the camera's objective focal axis).

FIG. 5 illustrates a small hand held controller 500 being held by aperson in an ultra-light aircraft 550 equipped with a multispectral dataacquisition system. This hand held controller 500 includes a videoscreen 312, a camera motion control interface 520 and a camera settingcontrol interface 530. The small hand held controller 500 enables anoperator to adjust remote sensing device settings (such as theemissivity setting of an IR camera) in real time as multispectral videosare being captured.

A suitable data acquisition system 100 preferably utilizes severalmass-produced, “off-the-shelf” components, because they tend to be morelight-weight, compact, and economical than custom-made components. Forexample, a suitable camera 115 or 120 comprises a broadband color CCDblock camera made by Sony® under the product designation “FCB-EX780SP.”This particular camera is operable to capture wavelengths from betweenabout 0.2 and about 0.9 μm in length, i.e., the visible spectrum and thenear infrared and near ultraviolet portions of the spectrum. Becausethis particular camera extends into the near ultraviolet portion of thespectrum, it is also suitable, if chilled and combined with a filter, tofunction as the ultraviolet imager 145. For example, a 0.24-0.28 μmwavelength pass-through filter would enable the camera to capture imagesof high-voltage coronas on electric distribution lines and components.

A suitable infrared imager 130 is sold under the trademark ThermoVisionRanger™ by Flir Systems, Inc.™ of Portland Oreg. This particular imager130 has a 3-5 μm near infrared spectral range. A suitable infraredimager 135 is sold under the trademark ThermoVision 2000™ by FlirSystems, Inc. This particular long-range, platform mounted imager has an8-9 μm far infrared spectral range.

The data acquisition system 100 obtains valuable 3-dimensionalinformation about the topography of a surface through its laser rangerfinder 140. A suitable laser range finder 140 is sold by Optech™ ofToronto, Canada, under the product designation “ALTM 30/70”. The laserrange finder 140 is used to obtain detailed and accurate elevation ordepth information about a topographic surface.

The data acquisition system 100 obtains important subsurface informationthrough its magnetometer system 150. A magnetometer is an instrument formeasuring the variations in the magnitude and direction of a magneticfield, and in particular, variations in the earth's magnetic field.Surface and subsurface features such as a buried pipeline can distortthe earth's magnetic field in ways that can be remotely sensed with ahandheld or aircraft-mounted magnetometer. A suitable magnetometer 150comprises a cesium magnetometer sensor and electronics package is soldby Geometrics™ of San Jose, Calif., under the product designation“G-822A”.

In a preferred embodiment, an array of at least two of these cesiummagnetometers are adjustably mounted on one or more booms attached to ahelicopter or ultralight aircraft at a distance of about 40 feet apart.At this distance, two linear data streams can be acquired during aflyover of a pipeline, eliminating the need, in some applications(especially pipeline applications), to make parallel flight lines overthe surveyed area. The magnetometers are adjustably mounted to maintainthe sensors' equators and long axes at angles of at least 6 degrees awayfrom the earth's field vector.

During operation, magnetic field data is acquired from bothmagnetometers simultaneously at an above-ground altitude of up to 30,000feet, but most preferably at about 500 to 2,000 feet. The absolutemagnetometer data of the two magnetometers are compared to determine thelateral horizontal gradient between the two magnetometers. Themagnetometer data is then used to generate a magnetometer survey map ofthe terrestrial surface. In pipeline survey applications, themagnetometer data is used to detect cathodic breakdown of the pipes.

It is expected and anticipated that many other “off-the-shelf”components exist or will become available with advances in technologythat would also be suitable for the data acquisition system 100described herein and possibly superior to the components describedherein. Therefore, it will be understood that the invention disclosed inthis application is by no means limited to the specific “off-the-shelf”components noted herein.

As shown and depicted in FIG. 1, the data acquisition system 100 alsocomprises a broadband laser emitter 125 utilizing a light source havinga range of wavelengths from 0.1 to 18 microns. A light source passesthrough a very precise prism mounted on a piezo-ceramic wafer thatvibrates at frequencies of up to 1 GHz, providing a linear sweep ofwavelengths, which can be time-calibrated, that are emitted by the laseremitter 125. The other remote sensing devices 110 may be utilized todetect reflected laser light originating from the broadband laseremitter 125. In this manner, the remote sensing devices 110 can functionas both passive and active remote sensors.

In alternative embodiments, the broadband laser emitter can be used inplace of a conventional laser range finder to generate athree-dimensional topographical map of a surface. Because control of thepiezo-ceramic wafer can be used to precisely alter the direction of thelaser beam of the broadband laser emitter 125, there is no need to pulsethe laser light source, as is done with conventional laser rangefinders.

Airborne-based embodiments of the data acquisition system 100 preferablyinclude means for mounting the remote sensing devices 110 on theaircraft and means for stabilizing the remote sensing devices 110 duringflight. A suitable gyroscope-stabilized aircraft mount for the remotesensing devices 110 is marketed by K-Hill, Inc., of Fort Worth Tex.under the Sky Pack™ trademark.

The data acquisition system 100 also comprises a time code generator 160and time base corrector 165 to synchronize the remote sensing devices110. Airborne-based embodiments of the data acquisition system 100preferably also include a global positioning satellite (GPS) receiver170 to provide information about the precise location of the aircraft.In these embodiments, the GPS data also serves as an input to the timecode generator 160. Airborne-based embodiments of the data acquisitionsystem 100 also preferably include an inertial reference system 175 torecord the roll, pitch and heading of the aircraft to determine itsorientation in space. Input from the GPS receiver 170 and inertialreference system 175 enables the image and data processor 190 tocorrelate spatial reference information with the frames and/or datastreams captured by the remote sensing devices 110.

One embodiment of the image and data processor 190 takes the form of astandard, commercially available computer, in combination with one ormore image processing video boards plugged into a hardware slot of thecomputer. The image and data processor 190 preferably includes amultispectral image mixer module 192 operable to stack images capturedfrom the different remote sensing devices 110. The image and dataprocessor 190 also preferably includes a module or application 194operable to automatically orthorectify and integrate topographicalimages into a composite image or mosaic. A preferred process oforthorectification is described further below. A suitableorthorectification and mosaic module or application 194 is sold underthe trademark ER Mapper™ by the Earth Resource Mapping company of SanDiego, Calif.

The image and data processor 190 also preferably includes a featureextraction module or application 196 operable to identify geometricfeatures or profiles in the captured images. Suitable techniques forfiltering the images to find anomalies include contrast enhancement,edge detection, two-dimensional fast Fourier transformations and inverseFourier transformations, image averaging, and de-interlacing to removefield-to-field jitter. The purpose of the filtering is to reduce noiseand draw out hidden details contained in the images to improve anomalydetection. A filtering technique suitable for one terrain, such as alush green field, many not be suitable for a different terrain, such asan arid terrain.

The feature extraction module 196 should also be operable tocomparatively analyze the geometric profile of an image captured in onefrequency or wavelength (e.g., the infrared spectrum) with the geometricprofile of an image captured in any of the other frequency or wavelength(e.g., the visible spectrum), detect differences between the geometricprofiles of the various spectrum-specific images, analyze thosedifferences in reference to a rule-based filter or framework (e.g., aset of thresholds), and tag or track those geometric profiles thatsurvive the filtering process of the rule-based framework. In thismanner, the feature extraction module 196 is operable to identifyfeatures or geometric profiles that are detectable in one or morespectrums but invisible in one or more other spectrums. In this manner,the feature extraction module 196 can also be tuned to identify veryspecific anomalies.

The data acquisition system 100 optionally also includes land or marinebased data acquisition modules, such as an electrical conductivity (orresistivity) imaging device to measure terrestrial resistance to currentflow. It is known, for example, that areas that are moist or compactedare more conductive than areas that are dry, less compacted, or whichcontain voids.

In general, the multispectral data acquisition system of the presentinvention is designed for high-speed recording and analysis of data fromseveral frequencies of the electromagnetic spectrum. The remote sensorsare preferably mounted on an aircraft such as a helicopter or ultralightaircraft. Alternatively, the remote sensors may be manually carried ormounted on a terrestrial, marine, or amphibious vehicle.

FIG. 9 illustrates one embodiment of a method of operating the dataacquisition system. In step 910 logistical planning is performed. One ofthe most important aspects of any given job is logistical planning. Thisinvolves determining the scope and location parameters of the jobbecause all calibrations and timing issues are likely to be affected orbased upon this information.

For some applications it is important to select a time window for dataacquisition that minimizes shadow differential signatures, moisture andhumidity variances, temperature variances, and the like. The selectedtime window should also take into account the fact that radiationreflectance and emission characteristics of objects on the surveyedterrain may vary at different times of the day and may affect thequality of the data acquired. Furthermore, it is desirable to selectflight times that coincide with observation windows during which GPSsatellites will provide the most reliable, error-minimizedtemporospatial information.

Before performing the data acquisition, it is also desirable to havegeological and botanical information of the target area. Thisinformation can often be determined by viewing a USGS digital terrain orelevation model or existing satellite and aerial imagery of that area.

In step 920, the remote sensing devices (cameras and detectors) arecalibrated and adjusted to optimize the sensitivity of the devices toone or more targeted phenomena. For example, the emissivity setting ofany IR imagers are adjusted to facilitate detection of the targetedphenomena. IR cameras with a floating raster are also calibrated oradjusted to target a specific radiation window of interest, so that thethermal characteristics of infrared imaged objects following outsidethis window of interest will appear saturated. Under Wein's DisplacementLaw, black body terrain objects which have temperatures ranging from −40F to +150 F (the range of natural landscape temperatures) have peakemissions from 8.6 to 12.4. Therefore, optic systems and detectors arematched for maximum sensitivity in this region in order to achievesignificant terrain component differentiation.

The spectral windows of one or more sensing devices (or of the laseremitter of a LIDAR system) are also tuned or filtered as needed tocapture a targeted range of electromagnetic phenomena. The sensingdevice's fields of view, exposure time, and gain levels are alsoadjusted as needed. The ranging and power levels of the laser andmagnetometers and the frequencies of scanners and oscillators are alsoadjusted as needed.

In step 930, the sensing devices are time calibrated. It is desirable torelate all of the data tracks to a single coordinated specific timeinterval. The cameras mounted on the data acquisition system shouldcapture image frames simultaneously. A digital time code generatorfurnishes regular time signals (60 per second) to logic and data controlmodules which record the signals on the recording apparatus, thusproviding a common frame of reference. Alternatively, time codeexcitation is done using the GPS signals as the excitation source fortiming.

In step 940, a flight is taken and images are acquired using the dataacquisition system. In the operation of an airborne-based embodiment ofa data acquisition system, an aircraft is flown at a constant altitudeabove sea level over the area to be surveyed. The sensing devices of thedata acquisition system are directed at the area and synchronizedrecordings are made of the signals from these devices. The airborneportion of the data acquisition system is carried in the cargo area ofthe aircraft with the equipment operator operating it by remote control.

The data acquisition system preferably captures, records and relates thefollowing data during flight: images and data from each of the sensingdevices and camera, a time code, GPS three-dimensional spatial data andinertial reference system data (such as speed and direction) for theaircraft, and the height above ground/target area data using a radaraltimeter or laser altimeter. The data acquisition system alsooptionally records camera setting data (such as the focal or field ofview data for one or more cameras) in order to facilitate an automatedorthorectification process.

During the data acquisition process, the operator (e.g., the pilot,navigator, or technician) pass over the target area and verballydocument landmark characteristics and other information onto the datastream. The data acquisition system records an audio of such remarksalong with the sensing device data, longitudinal and latitude data (inminutes and seconds) of the aircraft or image target, and otherinterpretive data. These activities facilitate accuracy.

In step 950, the operator views the image data and adjusts settings forthe remote sensing devices in real time. The operator is desirablytrained to make the proper kinds of adjustments in order to moresensitively capture the target data.

In step 960 the image data is orthorectified. The process oforthorectification 930, which can take many different forms, can becarried out in either two or three dimensions and is explored furtherbelow in conjunction with FIG. 10.

In step 970 images from different electromagnetic spectra are integratedand co-processed to identify and interpret anomalies. This process,which may also take many different forms, is explained further below inconjunction with FIG. 11.

FIG. 10 is a functional block diagram of one method of spatiallycoordinating or orthorectifying the visual image data.Orthorectification is performed to remove geometric distortionsintroduced by imperfect camera lens, such as spherical distortions fromusing a lens with a short focal length. The aim of theorthorectification process is to make any two overlapping pixels of two“overlapping” captured image frames to represent the same piece ofterrain. When completed, the orthorectified image data facilitatesautomatic computerized anomaly identification through comparativepixel-by-pixel analysis of the images.

As illustrated by step 1010, the process of orthorectification beginswith the identification or selection of a reference image. Then, in step1020, matching landmarks visible in both the captured and referenceimages are identified. Preferably at least ten correspondingorthorectification points on the captured and reference images areidentified. For illustration, FIG. 6 depicts a plurality ofcorresponding orthorectification points (by the letters A, B, C, and D)on a pre-orthorectified image 610 and a topographic map 620 chosen as areference image to which the image 610 is to be fitted. The process ofidentifying corresponding orthorectification points can be done manuallyor automatically using computer-executed geometric profile recognitionalgorithms. This process is facilitated if the reference image obtainedcomes pre-populated with orthorectification points or other data (e.g.,highway data) from which orthorectification points (e.g., intersections)can easily be identified or algorithmically extracted.

In step 1030, computer executed algorithms derive various mathematicaltransformation matrixes, that when multiplied by the captured imagedata, effectively rotate, stretch and bend the captured image until itsidentified landmarks perfectly align with the corresponding landmarksidentified in the reference image. Once orthorectified, the capturedimage spatially matches with and can be overlaid onto the referenceimage. FIG. 7 illustrates an infrared image that has been orthorectifiedto fit over a visible image slice.

Orthorectification is sometimes narrowly understood as a mathematicalprocess of removing the distortion caused by relief and the camerawithin a photograph so that the scale is uniform throughout the outputimage. In this sense, the process of orthorectification scales andaligns the image to a specified geodetic coordinate system. In someembodiments of the present invention, orthorectification is performed toa geodetic coordinate system. USGS topographical data may be used as areference image source and the captured images orthorectified to thattopographical data. In these embodiments, the reference image selectedin step 1010 comprises a three dimensional topographical map such as aUSGS GIS digital terrain or elevation model. In these embodiments, theprocess of orthorectification permits the captured and now georeferencedimages to inherit the elevation data of the topographical map. In theseembodiments, each georeferenced orthorectified image slice is preferablystored as a separate layer from the survey map data, and thegeoreferenced orthorectified images may then be rotated and viewed inthree dimensions.

But in other embodiments, orthorectification need not be performed toscale and align the captured images to a specified geodetic coordinatesystem. Instead, the reference image selected in step 1010 may compriseone of the captured images. The remaining remote sensing image data areorthorectified to the selected reference image.

An advantage of this alternative, non-geodetic orthorectification isautomation. A technician may be employed to identify the commonorthorectification points between multispectral images of a first set ofsimultaneously-captured frames. Software is then used to derive thetransformation matrixes needed to orthorectify each of the images to thechosen reference image. Because all of the cameras are mounted on acommon platform and run synchronously, this manualorthorectification-point-identification process need only be done once,with the first set of frames, provided that the cameras remain fixedrelative to each other and no changes are applied to the field of view(such as applying zoom) to any of the cameras. The transformationmatrixes derived to orthorectify the first set of overlapping imageframes may and are preferably reused, in automated fashion, toorthorectify subsequent frame sets.

Another method of orthorectification (not illustrated) is to use therecorded GPS spatial reference data, the recorded field of view data forthe camera that took the image, and any recorded altimeter data togeometrically calculate the spatial position of the center of the imageat the ground and the spatial extent of the boundaries of the capturedimage. This information can be used to either calculate the amount ofstretching, skew, and rotation needed to authomatically orthorectify theimage to two dimensions, or to facilitate identification ofcorresponding topographical map data (including correspondingorthorectification points). Yet another method of orthorectification(not illustrated) is to use an image manipulation algorithm toincrementally rotate, skew, filter and refilter the image data andcompare it with the topographical map until it finds a good match.

FIGS. 8A and 8B illustrate a series of logically stacked orthorectifiedimage slices simultaneously captured from different remote sensingdevices. An orthorectified visible light spectrum image 810 illustrateswhat the human eye would see of the surveyed terrain. Aspatially-corresponding orthorectified low-band infrared image 820illustrates radiation data for the surveyed terrain. The image 820 mayalso illustrate reflectivity and absorption data from a LIDAR system. Aspatially-corresponding orthorectified high-band infrared image 830illustrates radiation and density data for the surveyed terrain. Aspatially-corresponding orthorectified ultraviolet band image 840reveals the extent and health of photosynthesis taking place in plantlife below. A spatially-corresponding orthorectified three-dimensionallaser ranging image 850 provides positional and three-dimensional datafor the surveyed terrain. A spatially-corresponding orthorectifiedmagnetic data survey slice 860 is useful for identifying ferrous objectsand geo-magnetic data in the surveyed terrain. Most or all of theseorthorectified image slices can be three-dimensionally rotated andcomparatively filtered and analyzed to identify anomalies.

FIG. 11 illustrates an embodiment of a computer-automated process todigitally compare spatially correlated pixel data in the first andsecond images to identify anomalies and to digitally filter theanomalies to distinguish targeted anomalies from untargeted anomalies.This process works with a set of overlapping data slices comprising acombination of visible, infrared, and optionally also magnetometer andLIDAR elevation mapping data taken of a targeted terrestrial surfacearea to identify targeted hidden or invisible anomalies on or below thetargeted terrestrial surface area. It is assumed that the infrared andvisible images have already been orthorectified to each other or someother reference image. In step 1110, the pixel resolution of either thevisible image slice or the infrared image slice is adjusted throughpixel averaging until there is a one-to-one correspondence betweenpixels in the orthorectified image and infrared image slices.Non-overlapping remnants of the visible and infrared image slices arediscarded. Alternatively, step 1110 is done as a part of theorthorectification process.

In step 1120, each pixel of the orthorectified visible image slice isconverted to a single grayscale or intensity value. The visible imageslice is converted to a grayscale resolution (e.g., 6 bits or 64 shadesof gray) that matches the grayscale resolution of the infrared imageslice. The variables used for normalization and grayscale conversion ofthe different components of the color model represented by each colorimage pixel are determined using an iterative process or artificialintelligence techniques to create an optimally close match between thegrayscaled visible image slice and the infrared slice. Once thevariables for grayscale conversion are determined, the same variablesare used to convert each color pixel of the visible image frame tograyscale.

In one embodiment, an optimally close match between two images is one inwhich the greatest number of corresponding pixels have identicalintensity values. In another embodiment, an optimally close matchbetween two images is one in which the greatest number of correspondingpixels have nearly-equal intensity values. In yet another embodiment,identical pixel values are given one weight and near-equal intensityvalues are given weights of a lesser or greater value, and an optimallyclose match between two images is one which has the greatest or smallestsum of the weights.

Because the images are digitally represented, each image may berepresented as an array. In step 1130, visible and infrared differencearrays or difference images are generated in which corresponding pixelshaving identical intensity values are given a value of zero, and wherecorresponding pixels having non-identical intensity values retain theiroriginal values. Alternatively, visible and infrared difference arraysor difference images are generated where intensity differences ofcorresponding pixels less than a given threshold are given a value ofzero, and the remaining elements retain their original values. Non-zerovalues in the difference arrays identify potential anomalies. In yetanother alternative, a single difference array is generated in whicheach element has a value equal to the absolute value of the differencebetween the intensity values of the corresponding elements of thevisible and infrared image arrays.

Advantageously, the present invention includes a variety of automatedtechniques for filtering the difference arrays or images to filter theanomalies. In step 1140, standard pattern recognition techniques areused to filter anomalies in the difference images. For example, patternrecognition techniques may be used to identify straight-line anomaliescorresponding to a buried pipeline. In step 1150, the difference imagesare compared with magnetometer survey data to separate anomalies thathave also been picked up by the magnetometer survey from anomalies thathave not been picked up by the magnetometer survey. In step 1160, thedifference images are compared with LIDAR elevation profiles tointelligently separate surface anomalies from subsurface anomalies. Forexample, this comparison may be used to separate anomalies found on flator gently sloping surface portions from anomalies generated bybuildings, cars, and other structures on the surface.

Each of steps 1140, 1150, and 1160 are optional anomaly filteringtechniques. Use of any one or any combination of those steps isconsidered to fall within the scope of the present invention. In step1170, the entire process of steps 1130-1160 may optionally be repeatediteratively using different difference detection thresholds. In step1180, the filtered anomalies may be filtered yet again by comparingdifferent sets of frames to eliminate anomalies that do not persist inoverlapping portions of preceding or subsequent frames. This eliminatesnoisy anomalies. In step 1190, the remaining filtered anomalies arehighlighted on one or more of the original image slices. Furthermore,the anomalies may be highlighted on a series of frames to create a videoor animation that depicts the anomalies. Preferably, steps 1110-1190 arecomputer-automated, meaning that after initialization (which may involvesome human input), the steps are automatically performed withoutintensive ongoing human feedback to assist the image manipulation andpattern recognition processes. This is in contrast to using an imagemanipulation program like Adobe Photoshop® to manually andinconsistently manipulate images on a frame by frame basis without theassistance of macros or other automatic image rendering batch processes.

Although the foregoing specific details describe various embodiments ofthe invention, persons reasonably skilled in the art will recognize thatvarious changes may be made in the details of the apparatus of thisinvention without departing from the spirit and scope of the inventionas defined in the appended claims. Therefore, it should be understoodthat, unless otherwise specified, this invention is not to be limited tothe specific details shown and described herein.

1. A method of remotely sensing and identifying subsurfacecharacteristics of a terrestrial region, the method comprising: movingan aircraft equipped with a multispectral data acquisition system over aterrestrial region, said multispectral data acquisition systemincluding: a computer to receive, store, and process data; a firstimaging sensor communicatively coupled with said computer and operableto capture images in a visible band of the electromagnetic spectrum; asecond imaging sensor communicatively coupled with said computer andoperable to capture images in an infrared band of the electromagneticspectrum, and being mounted relative to the first imaging sensor so thatone of the two simultaneously-captured images from the first and secondimaging sensors will substantially overlap the other; a magnetometerarray to detect disturbances in the earth's magnetic field caused byobjects on or near the terrestrial surface, the magnetometer array beingin communication with said computer; a laser emitter and detector systemin communication with said computer to facilitate computation of thedistance between the laser emitter and a point on the terrestrial regionbelow, thereby to enable generation of a three-dimensional elevationmodel of the terrestrial region; a global positioning satellite receivertuned to receive positional data derived from global positioningsatellites, said global positioning satellite receiver beingcommunicatively coupled with said computer; capturing a first image withsaid first imaging sensor; simultaneously capturing a second image withsaid second imaging sensor; simultaneously capturing magnetic field datawith said magnetometer array; simultaneously surveying the terrestrialregion with the laser emitter and detector system; recording said firstimage, said second image, said magnetic field data, data from said laseremitter and detector system, and correlative spatial referenceinformation received from the global positioning satellite receiver;generating a magnetometer survey map with said magnetic field data;spatially correlating the first and second images and magnetometersurvey map with each other to enable spatially correlated comparisonsbetween the first image, second image, and magnetometer survey map;digitally comparing the spatially correlated first and second images toidentify anomalies; and digitally filtering said anomalies thatspatially correspond with recorded distortions in the magnetic fielddata.
 2. The method of claim 1, further comprising the step ofconverting the first image to a grayscale image.
 3. The method of claim1, further comprising the steps of: generating a digital elevation modelof the terrestrial region using the data recorded from said laserdetector and emitter system; spatially correlating the first and secondimages and digital elevation model with each other to enable spatiallycorrelated comparisons between the first image, second image, anddigital elevation model; digitally comparing the spatially correlatedfirst and second images to identify anomalies; and with the aid of saiddigital elevation model, digitally filtering said anomalies thatspatially correspond with flat or gently sloping areas of theterrestrial region.